Simulator, simulation method, and simulation program

ABSTRACT

The simulation reflects the actual behavior of a target in an application involving a target near a transporting surface of a carrier instead of being placed directly on the transporting surface. A simulator includes a creating unit that virtually creates a system in a three-dimensional virtual space, a tracking unit that updates positions of targets on the transporting surface in the three-dimensional virtual space based on a corresponding movement of the carrier, and updates a position of a target picked up by the processing device in association with a behavior of the processing device, and an instruction generation unit that generates a control instruction for the behavior of the processing device based on the position of each target. When the processing device places a target within a predetermined range from the transporting surface, the tracking unit associates the target with the transporting surface and updates a position of the target.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2015-225784 filed Nov. 18, 2015, the entire contents of which areincorporated herein by reference.

FIELD

The present invention relates to a simulator, a simulation method, and asimulation program for estimating the behavior of a system.

BACKGROUND

In the field of factory automation (FA), various automatic controltechniques are used widely. Such automatic control techniques may beused for applications for tracking the positions of targets such asworkpieces and processing the workpieces as intended, using variousrobots.

Designing or examining the system to be controlled with the aboveautomatic control technique may need preliminary evaluation of theperformance of the entire system. In response to this, a technique hasbeen developed for virtually creating a system and simulating itsoperation. For example, Japanese Unexamined Patent ApplicationPublication No. 2013-191128 (Patent Literature 1) describes a techniquefor integrated simulations of a mechanical system including a visualsensor in a real space corresponding to a virtual imaging unit. With thetechnique described in Patent Literature 1, a 3D simulator and a visualsensor simulator cooperate with each other to virtually generatecaptured images of a workpiece in a 3D space at predetermined timings.

Japanese Unexamined Patent Application Publication No. 09-258815 (PatentLiterature 2) describes simulations in which one model follows another.The simulations represent the kinetic chain of models that move inaccordance with the master-slave relationship defined between theaffecting model and the affected model.

CITATION LIST Patent Literature

Patent Literature 1: Japanese Unexamined Patent Application PublicationNo. 2013-191128

Patent Literature 2: Japanese Unexamined Patent Application PublicationNo. 09-258815

SUMMARY Technical Problem

In some applications, a workpiece may not be placed onto thetransporting surface of, for example, a conveyor, but may be releasedfrom a position a little away from the transporting surface and thenfall freely onto the transporting surface.

Although Patent Literature 2 describes a following operation performedin transporting a workpiece on a conveyor in FIG. 24, it neitherdescribes a method for defining the master-slave relationship betweenthe workpiece and the conveyor nor describes any processing for multipleworkpieces placed on the conveyor.

An application involving a target placed near a transporting surface ofa carrier instead of being placed directly on the transporting surfacemay also use simulations reflecting the actual behavior of the target.

Solution to Problem

A simulator according to one aspect of the present invention estimates abehavior of a system including a carrier having a transporting surfacefor continuously transporting a target that is placed thereon and aprocessing device for picking up and placing the target onto thetransporting surface. The simulator includes a creating unit, a trackingunit, and an instruction generation unit. The creating unit virtuallycreates the system in a three-dimensional virtual space. The trackingunit updates positions of a plurality of targets placed on thetransporting surface in the three-dimensional virtual space based on acorresponding movement of the carrier, and updates a position of atarget picked up by the processing device in the three-dimensionalvirtual space in a manner associated with a behavior of the processingdevice. The instruction generation unit generates a control instructionfor the behavior of the processing device based on the position of eachtarget in the three-dimensional virtual space. When the processingdevice places a target within a predetermined range from thetransporting surface, the tracking unit associates the target with thetransporting surface and updates a position of the target.

In some embodiments, when the processing device places a first targetwithin a predetermined range from a surface of a second targetdetermined to have been placed on a part of the transporting surface,the tracking unit associates a position of the first target and aposition of the second target with each other and updates the positionsof the first target and the second target.

In some embodiments, the tracking unit determines a size of thepredetermined range in accordance with a height of a target in thethree-dimensional virtual space.

In some embodiments, the simulator further includes an input unit thatreceives a setting for the size of the predetermined range. In thethree-dimensional virtual space, an area indicating the predeterminedrange is expressed by using a semitransparent object.

In some embodiments, when the processing device places a target within apredetermined range from the transporting surface, the tracking unitupdates a position of the target in a height direction in thethree-dimensional virtual space to bring the target into contact withthe transporting surface.

In some embodiments, when the processing device places a target within arange including the transporting surface and the second target, thetracking unit associates the target with one of the transporting surfaceand the second target based on predetermined priorities.

In some embodiments, the tracking unit corrects a position of a targetin a horizontal direction in the three-dimensional virtual space inaccordance with a transporting speed in the horizontal direction in thethree-dimensional virtual space at a timing when the processing devicereleases the target.

In some embodiments, the simulator further includes a measurement unitthat performs image measurement of an input image including at least apart of a target as a subject of the image in a manner associated withan area predefined on the transporting surface in the three-dimensionalvirtual space. In response to detection of a target by the measurementunit, the tracking unit displays the detected target in thethree-dimensional virtual space.

A simulation method according to another aspect of the present inventionis implemented by a computer for estimating a behavior of a system. Thesystem includes a carrier having a transporting surface for continuouslytransporting a target that is placed thereon and a processing device forpicking up and placing the target onto the transporting surface. Themethod includes a creating process, an updating process, and aninstruction generating process. The creating process includes virtuallycreating the system in a three-dimensional virtual space. The updatingprocess includes updating positions of a plurality of targets placed onthe transporting surface in the three-dimensional virtual space based ona corresponding movement of the carrier, and updating a position of atarget picked up by the processing device in the three-dimensionalvirtual space in a manner associated with a behavior of the processingdevice. The instruction generating process includes generating a controlinstruction for the behavior of the processing device based on theposition of each target in the three-dimensional virtual space. Theupdating process includes, when the processing device places a targetwithin a predetermined range from the transporting surface, associatingthe target with the transporting surface and updating a position of thetarget.

A simulation program according to another aspect of the presentinvention is used to estimate a behavior of a system including a carrierhaving a transporting surface for continuously transporting a targetthat is placed thereon and a processing device for picking up andplacing the target onto the transporting surface. The simulation programcauses a computer to implement a creating process, an updating process,and an instruction generating process. The creating process includesvirtually creating the system in a three-dimensional virtual space. Theupdating process includes updating positions of a plurality of targetsplaced on the transporting surface in the three-dimensional virtualspace based on a corresponding movement of the carrier, and updating aposition of a target picked up by the processing device in thethree-dimensional virtual space in a manner associated with a behaviorof the processing device. The instruction generating process includesgenerating a control instruction for the behavior of the processingdevice based on the position of each target in the three-dimensionalvirtual space. The updating process includes, when the processing deviceplaces a target within a predetermined range from the transportingsurface, associating the target with the transporting surface andupdating a position of the target.

Advantageous Effects

Embodiments of the present invention allow simulations reflecting theactual behavior of a target in an application involving a target placednear a transporting surface of a carrier instead of being placeddirectly on the transporting surface.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing the configuration of a system tobe simulated by a simulator according to one embodiment.

FIG. 2 is a schematic diagram describing a simulation method implementedby the simulator according to the embodiment.

FIGS. 3A and 3B are schematic diagrams describing examples ofapplications used by the simulator according to the embodiment.

FIG. 4 is a schematic diagram describing a landing detection height usedin the simulator according to the embodiment.

FIG. 5 is another schematic diagram describing the landing detectionheight used in the simulator according to the embodiment.

FIG. 6 is a schematic diagram showing the hardware configuration of thesimulator according to the embodiment.

FIG. 7 is a schematic diagram showing the functional structure of thesimulator according to the embodiment.

FIG. 8 is a flowchart showing the procedure of simulation performed bythe simulator according to the embodiment.

FIG. 9 is a flowchart showing the detailed procedure for the first halfpart of step S28 in FIG. 8.

FIG. 10 is a diagram showing one visualized system model created in athree-dimensional virtual space by the simulator according to theembodiment.

FIGS. 11A and 11B are schematic diagrams showing example user interfacescreens for setting a landing detection height used in the simulatoraccording to the embodiment.

FIG. 12 is a schematic diagram describing a landing detection rangedefined when the landing detection height is set to zero in thesimulator according to the embodiment.

FIGS. 13A and 13B are schematic diagrams describing a contact detectionmargin used in the simulator according to the embodiment.

FIG. 14 is a schematic diagram describing an overview of detection ofcontact with another workpiece in the simulator according to theembodiment.

FIG. 15 is a flowchart showing the detailed procedure for the secondhalf part of step S28 in FIG. 8.

FIGS. 16A to 16C are schematic diagrams describing associations betweenworkpieces in the simulator according to the embodiment.

FIG. 17 is a diagram showing visualizing simulation results obtainedfrom the simulator according to the embodiment.

FIG. 18 is a schematic diagram describing position corrections in aheight direction in the simulator according to the embodiment.

FIGS. 19A to 19C are schematic diagrams describing the process forcorrecting workpiece positions in the simulator according to theembodiment.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described in detailwith reference to the drawings. The same or corresponding components inthe figures are given the same reference numerals, and will not bedescribed redundantly.

A. Simulation

A simulation for estimating the behavior of a system performed by asimulator according to the present embodiment will now be described.

The simulator according to the present embodiment estimates (orsimulates) the behavior of a system including a carrier having atransporting surface for continuously transporting a target placed onthe surface, and a processing device for picking up the target andplacing the target onto the transporting surface.

In the present embodiment, one or more conveyors function as the carrierhaving a transporting surface for continuously transporting targetsplaced on the surface, and one or more robots function as the processingdevice for processing the targets. The carrier and the processing deviceare not limited to these examples and may be selected as appropriatedepending on the system to be simulated. The targets may hereafter alsobe referred to as workpieces. A workpiece may be any item that allowsposition tracking, and may for example be an end product or its part, oran intermediate product or its part.

The simulation performed by the simulator according to the presentembodiment will now be described briefly.

FIG. 1 is a schematic diagram showing the configuration of a system tobe simulated by the simulator according to the present embodiment. Asshown in FIG. 1, for example, a conveyor tracking system 1 includes arobot 210, which picks up a workpiece 232 within a predeterminedtracking area 231 when the workpiece 232 transported continuously on aconveyor 230 reaches the tracking area 231, and transfers to and placesthe workpiece 232 into a tracking area 235 on a conveyor 240. Thisseries of actions performed by the robot 210, or picking, transferring,and placing, may be referred to as the pick-and-place operation. Forease of explanation, a conveyor for transporting workpieces to be pickedup is also referred to as a workpiece pickup conveyor, whereas aconveyor on which the workpieces are to be placed is also referred to asa workpiece placement conveyor.

To enable the pick-and-place operation of the robot 210, an imaging unit222 captures an image of an imaging area 221 defined on a part of theconveyor 230, and a visual sensor 220 performs image measurementincluding pattern matching of an input image captured by the imagingunit 222 and obtains the measurement results including information aboutthe position, type, orientation, and other parameters of the workpiece232.

A controller 200 executes a predetermined control logic based on themeasurement result obtained by the visual sensor 220 to sequentiallyupdate (or track) the position of the workpiece 232 and generate acontrol instruction for the robot 210. The controller 200 typicallyincludes a programmable logic controller (PLC).

To generate the control instruction for the robot 210, the controller200 refers to the status value of the robot 210, and an encoder valueprovided from an encoder 236, which is connected to a drive roller 234for driving the conveyor 230 (encoder value 1), and an encoder valuefrom an encoder 246 coupled to a drive roller 244 for driving theconveyor 240 (encoder value 2). The controller 200 and the visual sensor220 are connected to each other with a network 202 to allow datacommunication between them. The measurement results are transmitted fromthe visual sensor 220 to the controller 200 through the network 202.

Evaluating the processing capability (e.g., a tact time) and theaccuracy of processing may be intended before the conveyor trackingsystem 1 shown in FIG. 1 is installed. This is because actuallyinstalling the conveyor tracking system 1 and checking the processingcapability is often difficult due to the cost or time constraints. Thesimulator according to the present embodiment is designed to achievesimpler estimation of the behavior of the conveyor tracking system 1.Thus, the simulator according to the present embodiment virtuallycreates a system to be simulated in a three-dimensional virtual space toachieve more efficient simulation.

FIG. 2 is a schematic diagram describing a simulation method implementedby the simulator according to the present embodiment. With reference toFIG. 2, the simulator models the entire conveyor tracking system 2,which is to be simulated, and incorporates an input image into thissystem model to simulate an image-capturing operation performed by theimaging unit 222.

The input image incorporated into the system model of the conveyortracking system 2 can represent specifications expected in the design(e.g., the moving speed of a workpiece 232 or the number of workpieces232 passing per unit time). Typically, the input image may be an imageactually captured on a similar production line.

Although the input image used in the simulation is expected to be animage captured in the existing system (e.g., the system before updatewhen an updated system is to be simulated), the input image may becaptured in any system and situation. More specifically, the input imagemay be any image including chronological change information about atarget to be simulated (typically, a workpiece 232).

The input image may be moving image data or data representing aplurality of still images arranged chronologically. The reproductionspeed of the moving image data or the update frequency of the datarepresenting the still images can be adjusted as appropriate to adjustthe chronological changes (or the moving speed) of a target to becontrolled. Adjusting the input image incorporated in the system modelin this manner allows the simulation to yield an optimal value for thechronological changes of the control target.

Additionally, still images that are not actually captured sequentiallybut are captured in different situations may be arranged aschronologically changing images and used as an input moving image.Although the images generated in this case have no workpieceoverlapping, this causes substantially no problem in performing thesimulation.

B. Overview

The processing for simulation performed by the simulator according tothe present embodiment will now be described briefly.

In applications involving the pick-and-place operation, a workpiece maybe released above the transporting surface of the conveyor, instead ofbeing released after placed onto the transporting surface. In otherwords, a robot may release a workpiece at a position above thetransporting surface of the conveyor. This intends to prevent theworkpiece from being damaged when contacting the conveyor.

FIGS. 3A and 3B are schematic diagrams describing examples ofapplications used by the simulator according to the present embodiment.With reference to FIG. 3A, a robotic arm 214 has a robot tool 216 on itsend to attract a workpiece. The robot tool 216 is, for example, an airsuction device, and produces a negative pressure to hold a workpiece.For example, the robot picks up a workpiece on the workpiece pickupconveyor (not shown), and operates its robot tool 216 to release theworkpiece at a position above the transporting surface of the workpieceplacement conveyor 240. The workpiece then falls under gravity and landson the transporting surface of the conveyor 240, and is transportedsubsequently.

One application may involve multiple processes in which differentworkpieces are placed. For this application, one workpiece is releasedabove another workpiece that has already been placed.

An application for placing a workpiece 232 onto a workpiece 242 that hasalready been placed will be described with reference to FIG. 3B. In thiscase as well, the workpiece 232 is released at a position above theworkpiece 242. The application shown in FIG. 3B may also be used forplacing multiple workpieces sequentially into a single container.

With simulation techniques known in the art, the workpiece released inthe application shown in FIG. 3B is not determined to be placed on theconveyor, because the workpiece is not on the transporting surface ofthe conveyor at the timing when the workpiece is released (or in otherwords when the workpiece is placed). The simulation cannot calculate theinfluence of gravity acting on the workpiece, and thus cannot beaccurate for the application shown in FIG. 3B.

To overcome this, the simulator according to the present embodiment cansimulate the behavior nearer the reality for an application in which aworkpiece is placed above the transporting surface of the carrier.

FIG. 4 is a schematic diagram describing a landing detection height usedin the simulator according to the present embodiment. With reference toFIG. 4, a predetermined range from the transporting surface of theconveyor 240 is defined as a landing detection height. When the robottool 216 arranged at the end of the robotic arm 214 releases theworkpiece 232 with its part being within the range of the landingdetection height, the workpiece 232 is determined to have landed on theconveyor 240. The workpiece 232 is then moved in synchronization withthe conveyor 240. The workpiece 232 attracted to the robot tool 216shown in FIG. 4 is immediately before being released by the robot tool216.

In this manner, the simulator according to the present embodiment mayset the landing detection height used for the workpiece placementconveyor 240. When the system model, which is a simulation target, isdisplayed in a three-dimensional virtual space, an object 250 indicatingthe area defined by the landing detection height may also appear. Theobject 250 may be semitransparent (or have a color with predeterminedtransmittance) so that it does not block view of the workpiece 232 inthe three-dimensional virtual space. More specifically, asemitransparent object may represent the area defined by the landingdetection height in the three-dimensional virtual space.

As shown in FIG. 4, when the pick-and-place operation causes thereleased workpiece 232 to enter the area indicated by the object 250,the workpiece 232 moves in synchronization with the advancing conveyor240, although the workpiece 232 is not in contact with the conveyor 240.

FIG. 5 is another schematic diagram describing the landing detectionheight used in the simulator according to the present embodiment. Asshown in FIG. 5, when multiple workpieces 242 and 232 are at leastpartially within the area defined by the landing detection height, theworkpieces 242 and 232 and the object may also move in synchronizationwith the advancing conveyor 240. The workpiece 232 attracted to therobot tool 216 shown in FIG. 5 is immediately before being released bythe robot tool 216.

As described above, when the robot places a workpiece within apredetermined range from the transporting surface, the simulator 100updates the position of the workpiece in a manner associated with thetransporting surface. This enables the simulation, without reflectingthe gravity acting on the workpiece, to be substantially the same as thesimulation reflecting the gravity actually acting on the workpiece whenthe workpiece is released at a position away from the transportingsurface of the carrier.

C. Hardware Configuration of Simulator

The hardware configuration of the simulator 100 according to the presentembodiment will now be described. The simulator 100 according to theembodiment is implemented typically by one or more computers executing aprogram.

FIG. 6 is a schematic diagram showing the hardware configuration of thesimulator 100 according to the present embodiment. With reference toFIG. 6, the simulator 100 is, for example, a computer having thegeneral-purpose computer architecture. The simulator 100 includes aprocessor 102, a main memory 104, an input unit 106, a display unit 108,a network interface 110, a hard disk drive (HDD) 120, an optical drive112, and a communication interface 116. These components are connectedto each other with an internal bus 118 in a communicable manner.

The processor 102 loads a program stored in the hard disk drive 120 intothe main memory 104 and executes the program to implement the functionsand the processing described later. The main memory 104 is a volatilememory and functions as a working memory used for program execution bythe processor 102.

The input unit 106 typically includes a keyboard, a mouse, a touchpanel, and a touchpad, and receives a user operation. The display unit108 includes a display and an indicator, and presents various pieces ofinformation to a user.

The network interface 110 exchanges data with external devices such as aserver through a network. The optical drive 112 reads various programsfrom an optical disc 114 or other media, and installs the programs intothe hard disk drive 120. The communication interface 116 is, forexample, a universal serial bus (USB) communication interface, andexchanges data with external devices such as an auxiliary storagethrough local communications.

The hard disk drive 120 stores an operating system (OS) 122, a programfor providing the functions of the simulator 100, such as a simulationprogram 124, and an image data group 140 including preliminarilyobtained input images used for simulation.

Although an intended program is installed in the simulator 100 via theoptical drive 112 in the configuration example shown in FIG. 6, aprogram may be downloaded from a server or other devices on the network.

When the simulator is a general-purpose computer as described above, anOS may be installed on the computer to provide the basic function of thecomputer, in addition to a program for providing the functions accordingto the present embodiment. In this case, the simulation programaccording to the present embodiment may call program modules included inthe OS in a predetermined order and/or at predetermined timings asappropriate to perform processing. More specifically, the programaccording to the present embodiment may not include these programmodules and may cooperate with the OS to perform processing. The programaccording to the present embodiment may not include such modules.

The program according to the present embodiment may be incorporated as apart of another program to be combined. The program according to thepresent embodiment may not thus include modules of the program to becombined, and may cooperate with the program to achieve processing. Inother words, the simulation program according to the present embodimentmay be incorporated in the other program.

Although FIG. 6 shows the simulator 100 that is a general-purposecomputer, the simulator 100 may be partly or entirely implemented usinga dedicated circuit (e.g., an application specific integrated circuit,or ASIC). Additionally, an external device may perform a part of theprocessing of the simulator 100.

Although the simulator 100 according to the present embodiment shown inFIG. 6 is a single device, two or more devices cooperating with eachother may function as the simulator 100. The simulator 100 according tothe present embodiment may include a system combining two or moreseparate devices.

D. Functional Structure of Simulator

The functional structure of the simulator 100 according to the presentembodiment will now be described.

FIG. 7 is a schematic diagram showing the functional structure of thesimulator 100 according to the present embodiment. The simulator 100shown in FIG. 7 includes a visual sensor simulator 150, a controllersimulator 160, a visualization module 170, a user interface module 180,and a system model emulator 190 as software functions. This group offunctional modules is typically implemented by the processor 102 (referto FIG. 6) executing the simulation program 124.

The user interface module 180 provides an operation screen for aidingthe user to set and create a setting parameter 152, a control program162, and three-dimensional design data 184. The user interface module180 also provides any user interface used when the visualization module170 displays simulation results.

The user interface module 180 includes a model creating module 182 forhandling the three-dimensional design data 184. The model creatingmodule 182 virtually creates the system to be simulated in athree-dimensional virtual space. More specifically, the model creatingmodule 182 displays a three-dimensional virtual space, and provides asetting and operation screen for creating the system to be simulated inthe three-dimensional virtual space.

The simulator 100 according to the present embodiment typicallyvirtually creates, in a three-dimensional virtual space, a systemincluding a carrier (typically, a conveyor) having a transportingsurface for continuously transporting a workpiece placed on the surfaceand a processing device for picking up the workpiece and placing theworkpiece onto the transporting surface. Further, as shown in FIG. 2,the system model further includes an imaging area 221 defined for thevisual sensor 220. The system model intended for the pick-and-placeoperation as shown in FIG. 2 may further define an area in which aworkpiece is to be picked up (tracking area 231) and an area in whichthe workpiece is to be placed (tracking area 235).

The visual sensor simulator 150 is a module for simulating theprocessing performed by the visual sensor 220, and performs imagemeasurement of an input image including at least a part of a workpieceas a subject of the image in a manner associated with the imaging area221 predefined on the transporting route (conveyor) in thethree-dimensional virtual space. More specifically, in response to afetch instruction (typically, a trigger signal) from the controllersimulator 160, the visual sensor simulator 150 retrieves thecorresponding image data from the preliminarily obtained image datagroup 140, and performs the image measurement in accordance with apredetermined setting parameter 152.

The measurement results from the image measurement performed by thevisual sensor simulator 150 are output to the controller simulator 160.The output processing corresponds to the transmission of the measurementresults obtained by the visual sensor 220 to the controller 200 throughthe network 202 in the conveyor tracking system shown in FIG. 1.

The controller simulator 160 performs a control operation for generatinga control instruction for the behavior of a robot, which is an exampleof the processing device, based on the measurement results from thevisual sensor simulator 150 and the position of each workpiece in thethree-dimensional virtual space. The controller simulator 160 is amodule for simulating the processing in the controller 200 (refer toFIG. 1), and performs a control operation (a sequence instruction, amotion instruction, or various functional instructions) in accordancewith the preliminarily created control program 162.

The control operation performed in the controller simulator 160 includesprocessing for generating a fetch instruction (trigger signal) forretrieving image data, which is to be transmitted to the visual sensorsimulator 150. More specifically, when a predetermined condition issatisfied, the controller simulator 160 generates a trigger signal. Thepredetermined condition is, for example, that the conveyor has moved bya predetermined distance, or a predetermined period has ended.

The system model emulator 190 sequentially updates the attributes ofeach object in the system model (e.g., the position, orientation, andspeed at each timing) as the simulation proceeds. More specifically, thesystem model emulator 190 creates and updates object attributeinformation 192 in a manner associated with each object included in thesystem model created by the model creating module 182.

The system model emulator 190 sets the positions and other attributes ofcomponents included in the created system model as initial values. Thesystem model emulator 190 updates the attribute values of each objectincluded in the object attribute information 192 as the simulationproceeds in accordance with a control instruction generated by thecontroller simulator 160. More specifically, the system model emulator190 is responsible for tracking, which updates the positions of multipleworkpieces placed on the transporting surface of the conveyor in thethree-dimensional virtual space based on a corresponding movement of theconveyor and also updates the position of a workpiece held by the robotin the three-dimensional virtual space in a manner associated with thebehavior of the robot.

The system model emulator 190 also outputs information indicating theposition or displacement of the conveyor to the controller simulator 160in a manner associated with the movement of the conveyor. In oneexample, the system model emulator 190 may output the encoder valueindicating a displacement from a reference position, or may generatepulses proportional to a movement of the conveyor per unit time. In thiscase, the encoder value indicates the position of the conveyor, and thenumber of pulses per unit time indicates the speed of the conveyor.

The controller simulator 160 outputs the time-series data for thecontrol instruction directed to the robot, and the trace data includingthe measurement results from the visual sensor simulator 150. The systemmodel emulator 190 outputs the trace data including the chronologicalchanges of each object in the system model.

The visualization module 170 uses the trace data to visualize theresults of the simulation performed for the system model. Morespecifically, the visualization module 170 uses the three-dimensionaldesign data 184, which is a definition file, to visualize the systemvirtually created in the three-dimensional virtual space, and also usesthe trace data provided from the controller simulator 160 to reproducethe chronological changes of the workpiece and the robot in the system.The simulation results may be visualized by the visualization module 170after a series of simulations is complete, or the simulation results maybe displayed as they are yielded during simulation.

In this manner, the visualization module 170 represents thechronological changes of the simulation results in the form of animationor a moving image on the display unit 108 of the simulator 100 (FIG. 6).

In the functional structure shown in FIG. 7, the controller simulator160 outputs the time-series data for its output control instruction tothe robot and also the trace data including the measurement results fromthe visual sensor simulator 150. However, the functional structure isnot limited to this example. The visualization module 170 may combinethe time-series data with the trace data to reproduce the systembehavior.

Although FIG. 7 shows the example in which the visualization module 170reproduces the behavior of the created system using the trace dataoutput from the controller simulator 160, the simulator 100 may notinclude the visualization module 170. For example, the trace data fromthe controller simulator 160 may be output to an external device or anexternal application, and the external device or the externalapplication may reproduce the behavior of the system. In someembodiments, the visualization module 170 may simply generate and storemoving image data for reproducing the behavior of the system in anystorage medium, which may then be reproduced by another application.

E. Procedure

The procedure of simulation performed by the simulator 100 according tothe present embodiment will now be described.

FIG. 8 is a flowchart showing the procedure of simulation performed bythe simulator 100 according to the present embodiment. With reference toFIG. 8, the simulator 100 first receives the settings of the systemmodel (step S2). The settings of the system model include thearrangement of the devices included in the system, and the moving speedof the conveyor, which is a carrier. Based on these system modelsettings, the simulator 100 (model creating module 182) virtuallycreates a system to be simulated (system model) in a three-dimensionalvirtual space.

The simulator 100 (user interface module 180) receives an imaging areafor a visual sensor defined in the system model (step S4). Based on therelative positional relationship between the created system and thedefined imaging area, the simulator calculates a calibration parameter,which is a conversion parameter for transforming the measurement resultsinto an input value for a control operation.

The simulator 100 (user interface module 180) then receives a controlprogram for controlling the system model (step S6). The control programis associated with the system, and is to be executed by the controllersimulator 160.

The simulator 100 (user interface module 180) receives the settings forimage measurement to be performed by the visual sensor simulator 150(step S8). The settings include designation of the processing details ofthe image measurement and reference information (e.g., a model image,and a feature quantity calculated from the model image) associated withthe designated processing details.

This procedure completes the settings for the simulation.

When instructed to start the simulation, the simulator 100 (system modelemulator 190) sets the attributes of each object included in the systemmodel as initial values (step S10). More specifically, the simulator 100(system model emulator 190) sets the initial values for the position ofeach workpiece included in the system model, the encoder valueindicating the position or movement of the conveyor or the position ofthe robot.

The simulator 100 (system model emulator 190) then updates the encodervalue of the conveyor to the predetermined value corresponding to theend of the first cycle of simulation, and also updates the position ofeach workpiece placed on the transporting surface of the conveyor (stepS12). Workpieces associated with each other (described later) areupdated equivalently.

The simulator 100 (controller simulator 160) then determines whether acondition for generating a trigger signal is satisfied based on theupdated encoder value and the position of each workpiece (step S14).

When the condition is satisfied (Yes in step S14), the simulatorvirtually generates a trigger signal (step S16). In response to thegenerated trigger signal, the simulator 100 (visual sensor simulator150) retrieves the corresponding image data from the preliminarilyobtained image data group, and performs the image measurement (stepS18).

The simulator 100 (system model emulator 190) determines whether a newworkpiece has been detected through the image measurement (step S20).When a new workpiece has been detected through the image measurement(Yes in step S20), the simulator 100 (system model emulator 190)generates a new object for the detected workpiece (step S22). In otherwords, the simulator 100 (system model emulator 190) displays thedetected workpiece in the three-dimensional virtual space in response tothe workpiece detection by the visual sensor simulator 150. The positionof the generated new object is calculated based on the position of theimaging area and the local coordinates included in the measurementresults.

When no new workpiece has been detected through the image measurement(No in step S20), the processing in step S22 is skipped.

When the condition for generating a trigger signal is not satisfied (Noin step S14), the processing in steps S16 to S22 is skipped.

Subsequently, the simulator 100 (controller simulator 160) performs acontrol operation in accordance with the control program 162 based onthe updated encoder value and the position of each workpiece to generatea control instruction (step S24). The simulator 100 (system modelemulator 190) updates the positions of the robot and the correspondingworkpiece based on the generated control instruction (step S26). Forexample, when the robot picks up or transfers any workpiece, theposition of the target workpiece is updated in accordance with thebehavior of the robot. When the robot places the workpiece (or forexample releases the workpiece), the position and the status of thetarget workpiece are updated.

The simulator 100 (system model emulator 190) then determines whetherthe workpiece has landed on the transporting surface of the conveyor orhas contacted another workpiece based on the updated position of theworkpiece (step S28). When the simulator determines that the workpiecehas landed on the transporting surface of the conveyor or has contactedanother workpiece (Yes in step S28), the simulator 100 (system modelemulator 190) associates the target workpiece with the conveyor, orassociates the target workpiece with the other workpiece (step S30).

The simulator 100 determines whether a preset simulation period hasended (step S32). When the simulation period has not ended (No in stepS32), the processing in step S12 and subsequent steps is repeated.

When the preset simulation period has ended (Yes in step S32), thesimulation completes.

After the completion of the simulation, the simulator may use the tracedata resulting from the simulation to visualize the behavior of thesystem.

F. Detecting Workpiece Landing on Transporting Surface

The processing for determining whether a workpiece has landed on thetransporting surface of the conveyor (processing in the first half ofstep S28 in FIG. 8) will now be described in detail.

FIG. 9 is a flowchart showing the detailed procedure for the first halfpart of step S28 in FIG. 8. With reference to FIG. 9, the simulator 100(controller simulator 160) determines whether a control instruction hasbeen generated for the robot tool to release a workpiece (step S2801).When the control instruction has not been generated (No in step S2801),the simulator 100 determines that no workpiece, following the robot tooland released, has landed on the transporting surface of the conveyor(step S2802). The processing then returns to step S28 and subsequentsteps in FIG. 8.

When a control instruction has been generated for the robot tool torelease a workpiece (Yes in step S2801), the simulator 100 obtains thecoordinates indicating the bottom position of the workpiece releasedfrom the robot tool in response to the control instruction (step S2803),and calculates the distance from the conveyor to the bottom of theworkpiece (e.g., the distance d in FIG. 4) (step S2804).

The simulator 100 determines whether the calculated distance from theconveyor to the bottom of the workpiece is equal to or less than apredetermined landing detection height (step S2805). When the calculateddistance is greater than the predetermined landing detection height (Noin step S2805), the simulator 100 determines that no workpiece,following the robot tool and released, has landed on the transportingsurface of the conveyor (step S2806). The processing advances to stepS2811 and subsequent steps shown in FIG. 15, which will be describedlater.

When the calculated distance is equal to or less than the predeterminedlanding detection height (Yes in step S2805), the simulator 100determines that the target workpiece has landed on the transportingsurface of the conveyor (step S2807). The processing then returns tostep S28 and subsequent steps in FIG. 8.

FIG. 10 is a diagram showing one visualized system model created in athree-dimensional virtual space by the simulator 100 according to thepresent embodiment. The simulator 100 according to the presentembodiment can render a model within a three-dimensional virtual spacein any direction. More specifically, a user can freely change the anglefrom which the visualized system model is viewed.

With reference to FIG. 10, the conveyor 230 transporting a workpiece tobe picked up and the conveyor 240 transporting a workpiece placed on itare arranged in parallel. The conveyors 230 and 240 are associated withtwo robots 311 and 313. In this system model, a workpiece 232 istransported by the conveyor 230 from left to right in the drawing. Whenthe workpiece 232 reaches the predetermined tracking area 231 or 233,the robot 311 or the robot 313 picks up the incoming workpiece 232 andplaces the workpiece onto the conveyor 240. The robots 311 and 313 eachplace the workpiece 232 in the corresponding tracking area 235 or 237defined on the conveyor 240. Each workpiece 232 placed in a randomorientation on the conveyor 230 is aligned in a predetermined directionwhen placed onto the conveyor 240.

In one example application, the conveyor 230 may transport at least twotypes of workpieces 232. The robot 311 is controlled to pick up andplace one specific type of workpiece, whereas the robot 313 iscontrolled to pick up and place another type of workpiece. The differenttypes of workpieces may have different shapes. In this case, a robothaving a special robot tool dedicated to a particular type of workpiecemay be used for that type of workpiece.

In the three-dimensional virtual space, the object 250 is superimposedon the conveyor 240. The object 250 is semitransparent (or has a colorwith predetermined transmittance) and indicates an area defined by thelanding detection height. The object 250 may be hidden from view by asetting operation performed by the user. When the object 250 appears,the user can readily check the range of the landing detection height inthe system model.

The landing detection height may be set freely in accordance with thesystem (application) to be simulated. In this case, the simulator 100may include an input unit for receiving the setting of the landingdetection height (the size of the predetermined range within whichworkpiece landing is to be detected).

FIGS. 11A and 11B are schematic diagrams showing example user interfacescreens for setting the landing detection height in the simulator 100according to the present embodiment. With reference to FIG. 11A, asetting entry screen 300 has an entry field 302, into which the landingdetection height is to be entered. The user can enter an intended valuedirectly or by operating a button 304.

Further, landing detection heights may be set separately for differenttypes (e.g., different items or different shapes) of workpieces. Asetting entry screen 310 shown in FIG. 11B has an entry field 312 for aworkpiece of Type 1, and an entry field 316 for a workpiece of Type 2.The user can enter an intended landing detection height in the entryfield 312 directly or by operating a button 314, and for the entry field316 directly or by operating a button 318.

When the landing detection height is set to zero, some margin may be setto compensate for detection errors.

FIG. 12 is a schematic diagram describing a landing detection rangedefined when the landing detection height is set to zero in thesimulator 100 according to the present embodiment. As shown in FIG. 12,an area within which workpiece landing is detected at a predeterminedheight (corresponding to the object 250 shown in FIG. 10) is definedabove the conveyor 240 (workpiece placement conveyor), although thelanding detection height is set to zero. The height of this area isdetermined in accordance with the accuracy in position management forthe system model.

The margin is set for the landing detection height to prevent anerroneous determination in determining whether the workpiece hascontacted the conveyor (contact detection), or specifically to prevent aworkpiece released from the robot at a position nearly contacting theconveyor from being determined not to contact the conveyor.

G. Detecting Contact with Another Workpiece

The processing for determining whether a workpiece has contacted anotherworkpiece (processing in the second half part of step S28 in FIG. 8)will now be described in detail. As described with reference to FIG. 3B,one workpiece may be placed on another workpiece that has already beenplaced on the transporting surface.

FIGS. 13A and 13B are schematic diagrams describing a contact detectionmargin used in the simulator 100 according to the present embodiment.With reference to FIG. 13A, the workpiece 232 placed on the transportingsurface of the conveyor has a contact detection margin surrounding theworkpiece 232 as a tolerance for detecting contact with anotherworkpiece. When another workpiece enters an area 260 defined by thecontact detection margin, the two workpieces are associated with eachother and are moved in synchronization with the conveyor. In the samemanner as for the object 250 representing an area defined by the landingdetection height shown in FIG. 10, the area 260 may be visualized byusing a semitransparent object. The visualization allows the user toreadily check the area defined by the contact detection margin.

As described above, the defined contact detection margin surrounds theworkpiece 232. In a system for placing multiple workpieces in a singlecontainer, for example, a target workpiece may not be placed exactly onthe previously placed workpiece but may simply be placed near theprevious workpiece. In that system, a workpiece 232 may have a contactdetection margin on its periphery in addition to its top surface. In asystem in which workpieces 232 are oriented randomly on the transportingsurface of the conveyor, a workpiece 232 may have a contact detectionmargin along the entire periphery, rather than having a contactdetection margin only on its particular surface. In an application forstacking workpieces on one another, a workpiece 232 may have a contactdetection margin only on its top surface.

FIG. 13B is a cross-sectional view taken along line A-A in FIG. 13A. Thesize of the detection area including the contact detection margin (areaheight Hm) may be determined freely or may be determined in accordancewith the height of the workpiece 232. For example, the contact detectionmargin may be determined in accordance with the height of the workpiece232 within a three-dimensional virtual space. For example, the areaheight Hm can be calculated by multiplying the height of the workpiece232 by a constant using the formulas below.Area height Hm=α×workpiece height hContact detection margin=(α−1)×workpiece height hwhere α>1 (const)

The user may also freely set the contact detection margin in the samemanner as for the user interface screen shown in FIGS. 11A and 11B.

Further, contact detection margins may be set separately for differentsurfaces of a workpiece, or for different types (e.g., different itemsor different shapes) of workpieces.

FIG. 14 is a schematic diagram describing an overview of detection ofcontact with another workpiece in the simulator according to the presentembodiment. With reference to FIG. 14, when the robot tool 216 arrangedat the end of the robotic arm 214 releases the workpiece 232, thedistance to the workpiece 242 placed under the workpiece 232 iscalculated. The workpiece 232 entering the area 260 of the workpiece 242defined by the contact detection margin is determined to be in contactwith the workpiece 242.

With one example procedure, distances d1 and d2 from the center ofgravity of the workpiece 232 are calculated. The distance d1 is from thecenter of gravity of the workpiece 232 to the workpiece 242 immediatelybelow the center of gravity. The distance d2 is from the center ofgravity of the workpiece 232 to the bottom of the same workpiece 232.When the difference between the distances d1 and d2 is equal to or lessthan the contact detection margin, the workpiece 232 is determined to bein contact with the workpiece 242. In this case, the workpieces 232 and242 are moved as a single unit in synchronization with the conveyor 240.

FIG. 15 is a flowchart showing the detailed procedure for the secondhalf part of step S28 in FIG. 8. With reference to FIG. 15, thesimulator 100 determines whether any workpiece has been placedimmediately below the workpiece released from the robot in response tothe control instruction (step S2811). When no such workpiece has beenplaced (No in step S2811), the simulator 100 determines that theworkpiece released from the robot in response to the control instructionis not in contact with another workpiece (step S2812). The processingthen returns to step S28 and subsequent steps in FIG. 8.

When any workpiece has been placed immediately below the workpiecereleased from the robot in response to the control instruction (Yes instep S2811), the simulator 100 calculates the distance d1 from thecenter of gravity of the target workpiece to the workpiece immediatelybelow the target workpiece (step S2813), and also calculates thedistance d2 from the center of gravity of the target workpiece to thebottom of the same target workpiece (step S2814). The simulator 100 thendetermines whether the difference between the distances d1 and d2 isequal to or less than the contact detection margin (step S2815).

When the difference between the distances d1 and d2 is equal to or lessthan the contact detection margin (Yes in step S2815), the simulator 100determines that the target workpiece is in contact with anotherworkpiece (step S2816). The simulator 100 corrects the position of thetarget workpiece to place the target workpiece on the workpieceimmediately below the target workpiece (step S2817). More specifically,the position of the workpiece 232 is updated to eliminate a certaindistance left between the workpiece 232 and the workpiece 242 as shownin FIG. 14. The processing then returns to step S28 and subsequent stepsin FIG. 8.

When the difference between the distances d1 and d2 is greater than thecontact detection margin (No in step S2815), the simulator 100determines that the workpiece released from the robot in response to thecontrol instruction is not in contact with another workpiece (stepS2812). The processing then returns to step S28 and subsequent steps inFIG. 8.

Through the processing described above, the simulator determines whetherone workpiece has contacted another workpiece. When determining that oneworkpiece has contacted another workpiece, the simulator places thetarget workpiece on the other workpiece, and these workpieces move insynchronization with the conveyor.

FIGS. 16A to 16C are schematic diagrams describing associations betweenworkpieces in the simulator according to the present embodiment. Withreference to FIG. 16A, when the workpiece 1 is associated with theworkpiece 2, the positions of the two workpieces undergo the samevariations. In the example shown in FIGS. 16A to 16C, the workpiece 1 isat the coordinates (x1, y1, z1) and the workpiece 2 at the coordinates(x2, y2, z2) at a given timing. As shown in FIG. 16B, an event causesthe coordinates of the workpiece 1 to change from (x1, y1, z1) to(x1+Δx, y1+Δy, z1+Δz). In this case, the coordinates of the workpiece 2undergo the same variation (Δx, Δy, Δz). As a result, the coordinates ofthe workpiece 2 change from (x2, y2, z2) to (x2+Δx, y2+Δy, z2+Δz) asshown in FIG. 16C.

As described above, when the robot places the second workpiece in apredetermined range (contact detection margin) from the surface of thefirst workpiece determined to have been placed on any transportingsurface, the simulator 100 according to the present embodiment updatesthe positions of the first workpiece and the second workpiece in amanner associated with each other. Through the processing shown in FIGS.16A to 16C for the associated workpieces, the multiple workpieces can bemoved as a single unit.

H. Priority for Associations Between Objects

As described above, the simulator 100 according to the presentembodiment can update the position of each workpiece in a mannerassociated with the transporting surface, or in a manner associated withanother workpiece. However, at the timing of landing detection andcontact detection, one workpiece determined to have landed on thetransporting surface can possibly be determined to have contactedanother workpiece. In this case, the priorities may be assigned toobjects with which the target workpiece is to be associated.

For example, when the workpiece 232 determined to have landed on theworkpiece placement conveyor 240 is also determined to have contactedanother workpiece placed on the workpiece placement conveyor 240, theworkpiece 232 may be associated with the conveyor 240 with a higherpriority, or the workpiece 232 may be associated with the otherworkpiece with a higher priority than with the conveyor 240.

The priority may be set to 1 for the workpiece placement conveyor 240,and may be set to 2 for the other workpiece. When multiple objects aredetected in the landing detection and the contact detection, an objecthaving the higher priority may be selected. This structure prevents theworkpiece from being associated with an unintended object.

In this manner, when the robot 210 places a workpiece in an areaincluding both the transporting surface and another target, thesimulator 100 according to the present embodiment can associate theworkpiece with either the transporting surface or the other workpiece inaccordance with their predefined priorities.

I. Updating Workpiece Position

The processing for updating the workpiece position in the simulator 100according to the present embodiment will now be described. In thesimulator 100 according to the present embodiment, the visualizationmodule 170 (FIG. 7) performs image measurement to search an input imagefor a workpiece, and the detected workpiece appears in athree-dimensional virtual space.

The measurement results of image measurement include the coordinates (x,y) indicating the center of a part (object) detected in the input image.The coordinates (x, y), which are values in a local coordinate systemused for image measurement, are to be transformed into the coordinatesin a three-dimensional virtual space.

More specifically, the simulator 100 can use transform coefficients A toG for transforming the coordinates (x, y) of an input image defined inthe camera coordinate system used in image measurement into thecoordinates (X, Y, Z) defined in a world coordinate system defining thethree-dimensional virtual space. The simulator 100 can thus calculatethe initial position at the input into the controller simulator 160based on the workpiece coordinates (x, y) detected in the visual sensorsimulator 150 in the manner described below.Workpiece initial position X0=A×x+B×y+CWorkpiece initial position Y0=D×x+E×y+FWorkpiece initial position Z0=G

A movement Xd of the conveyor in X-direction, a movement Yd of theconveyor in Y-direction, and a movement Zd of the conveyor (typically,zero) in Z-direction per pulse of an encoder value can be used tocalculate the workpiece position corresponding to a displacement Etindicated by the encoder value as written in the formulas below.Workpiece position (X)=Xd×Et+X0Workpiece position (Y)=Yd×Et+Y0Workpiece position (Z)=Zd×Et+Z0

When the absolute value of an encoder value is used, a deviation fromthe encoder value for each workpiece displayed initially may beincorporated in these formulas. The simulator 100 sequentially updatesthe position of each workpiece in accordance with these formulas.

J. Additional Objects

In the application described above, multiple workpieces are associatedwith each other and moved in synchronization with the conveyor. Specificexamples of this application include placing multiple workpieces in asingle container. In visualizing the simulation of this application, thecontainer may also be displayed.

FIG. 17 is a diagram showing visualizing simulation results obtainedfrom the simulator 100 according to the present embodiment. Withreference to FIG. 17, multiple objects 270 corresponding to containersare displayed on the conveyor 240 at intervals preset by the user. Theintervals between the objects 270 are, for example, preset based onrequired specifications (including a tact time). When simulation resultsare reproduced, the objects 270 appear on the conveyor 240 at the presetintervals.

The positions of the objects 270 may serve as references used forpositioning workpieces. In addition to the objects 270, reference lines280 may also be displayed.

The shape, color, and size of the objects 270 may be freely preset bythe user. Further, the objects 270 may be transparent or semitransparentto allow the user to readily check the positional relationship betweenthe objects 270 and the workpieces 232.

As shown in FIG. 17, the additional objects 270 and/or the referencelines 280 are virtually displayed in addition to the workpieces 232 toallow the user to visually evaluate whether the workpieces are processedas designed.

K. Correcting Workpiece Position in Height Direction

Correcting the position of a workpiece in the height direction will nowbe described. As described with reference to FIGS. 4 and 5, when aworkpiece is released within the range of the landing detection heightfrom the transporting surface of the conveyor, the workpiece isdetermined to have landed on the conveyor and then moved insynchronization with the conveyor.

The workpiece may retain its height after released, or may be correctedin the height direction to place the workpiece on the transportingsurface of the conveyor.

FIG. 18 is a schematic diagram describing position corrections in theheight direction in the simulator according to the present embodiment.With reference to FIG. 18, when the workpiece 232 is released at orbelow the landing detection height from the conveyor, its position inthe height direction is corrected to place the workpiece on thetransporting surface of the conveyor. The workpiece is then moved insynchronization with the conveyor 240.

As shown in FIG. 18, when the robot places the workpiece within therange of the landing detection height from the transporting surface ofthe conveyor, the simulator 100 (system model emulator 190 in FIG. 7)may update the position of the workpiece in the height direction in thethree-dimensional virtual space to bring the workpiece in contact withthe transporting surface. Correcting the workpiece position in theheight direction allows simulation of the behavior nearer the reality.This position correction may also be used for simulating stacking ofmultiple workpieces.

L. Correcting Workpiece Position Based on Robot Moving

Correcting the position of a workpiece based on the behavior of therobot during placement of the workpiece will now be described. In anactual pick-and-place operation, the robot moves horizontally at themoving speed of the conveyor on which the workpiece is to be placed,immediately before placing the workpiece. In other words, the robotreleases the workpiece after minimizing the relative velocity of therobot and the conveyor to nearly zero.

FIGS. 19A to 19C are schematic diagrams describing the process forcorrecting workpiece positions in the simulator 100 according to thepresent embodiment. With reference to FIG. 19A, the robot moveshorizontally in synchronization with the moving speed of the conveyor240 immediately before placing the workpiece (at time T1).

As shown in FIG. 19B, immediately after the robot tool 216 arranged atthe end of the robotic arm 214 releases the workpiece (at T1+Δt), theworkpiece is expected to have moved horizontally by a certain distance(ΔL) under the inertial force acting from the preceding horizontalmovement of the robot.

Thus, the position of the workpiece released from the robot may becorrected using the horizontal offset (distance ΔL), and then thesimulator may determine whether the workpiece has landed on thetransporting surface of the conveyor. More specifically, the simulator100 (system model emulator 190 shown in FIG. 7) may correct theworkpiece position horizontally in the three-dimensional virtual spacein accordance with the velocity of the robot in the horizontal directionin the three-dimensional virtual space when the robot releases theworkpiece.

As shown in FIG. 19C, immediately after the robot tool 216 arranged atthe end of the robotic arm 214 releases the workpiece (at T1+Δt), theworkpiece is precisely expected to move horizontally by a certaindistance (ΔL) under the inertial force acting from the precedinghorizontal movement of the robot and move vertically by a certaindistance (ΔH) under its weight.

Thus, the position of the workpiece when released from the robot may becorrected using the horizontal offset (distance ΔL) and the verticaloffset (distance ΔH), and then the simulator may determine whether theworkpiece has landed on the transporting surface of the conveyor.

For the correction using the offsets shown in FIGS. 19B and 19C, theoffsets may be variable and calculated based on the preceding movingspeed of the robot (and the acceleration as appropriate). To reduce thecomputational complexity, the offsets may be predetermined fixed values.

The correction enables the simulation reflecting the behavior nearer thereality.

M. Other Embodiments

In the above embodiment, the robot picks up a workpiece continuouslytransported on the workpiece pickup conveyor, transfers the workpiece tothe workpiece placement conveyor, and places the workpiece onto theworkpiece placement conveyor. The embodiment is not limited to thisstructure in which workpieces to be picked up are transportedcontinuously on the workpiece pickup conveyor. The embodiment alsocovers an application in which the robot picks up a designated workpieceor any workpiece from multiple workpieces that are stationary (e.g.,multiple workpieces stacked on one another).

N. Advantages

The simulator 100 according to the embodiments allows a workpiece to betracked correctly in synchronization with movement of the conveyor in,for example, the conveyor system with the pick-and-place operation withwhich a workpiece may be released above the transporting surface of theconveyor and placed onto the transporting surface.

The simulator 100 according to the embodiments further allows workpiecesto be associated with each other and tracked correctly insynchronization with movement of the conveyor in the conveyor system inwhich one workpiece is already placed on the transporting surface of theconveyor and another workpiece is released from the robot at a positionnear the already placed workpiece.

The embodiments disclosed herein should be considered to be in allrespects illustrative and not restrictive. The scope of the presentinvention is determined not by the description given above but by theclaims, and is construed as including any modification that comes withinthe meaning and range of equivalency of the claims.

REFERENCE SIGNS LIST

-   1, 2 conveyor tracking system-   232, 242 workpiece-   100 simulator-   102 processor-   104 main memory-   106 input unit-   108 display unit-   110 network interface-   112 optical drive-   114 optical disc-   116 communication interface-   118 internal bus-   120 hard disk drive-   122 OS-   124 simulation program-   140 image data group-   150 visual sensor simulator-   152 setting parameter-   160 controller simulator-   162 control program-   170 visualization module-   180 user interface module-   182 model creating module-   184 three-dimensional design data-   190 system model emulator-   192 object attribute information-   200 controller-   202 network-   210, 311, 313 robot-   214 robotic arm-   216 robot tool-   220 visual sensor-   221 imaging area-   222 imaging unit-   230, 240 conveyor-   231, 233, 235 tracking area-   234, 244 drive roller-   236, 246 encoder-   250, 270 object-   260 area-   280 reference line

The invention claimed is:
 1. A simulator for estimating a behavior of asystem comprising a carrier having a transporting surface forcontinuously transporting a target that is placed thereon and aprocessing device for picking up and placing the target onto thetransporting surface, the simulator comprising: a processor configuredwith a program to perform operations comprising: operation as a creatingunit configured to virtually create the system in a three-dimensionalvirtual space; operation as a tracking unit configured to updatepositions of a plurality of targets placed on the transporting surfacein the three-dimensional virtual space based on a corresponding movementof the carrier, and update a position of the target picked up by theprocessing device in the three-dimensional virtual space in a mannerassociated with a behavior of the processing device; and operation as aninstruction generation unit configured to generate a control instructionfor the behavior of the processing device based on a position of eachtarget of the plurality of targets in the three-dimensional virtualspace, wherein the processor is configured with the program to performoperations such that: in response to the processing device placing thetarget within a predetermined range from the transporting surface,operation as the tracking unit comprises operation as the tracking unitconfigured to associate the target with the transporting surface andupdate the position of the target and operation as the tracking unitcomprises operation as the tracking unit configured to correct theposition of the target in a horizontal direction in thethree-dimensional virtual space in accordance with a transporting speedin the horizontal direction in the three-dimensional virtual space at atiming in response to the processing device releasing the target.
 2. Thesimulator according to claim 1, wherein the processor is configured withthe program to perform operations such that, in response to theprocessing device placing a first target within a predetermined rangefrom a surface of a second target determined to have been placed on apart of the transporting surface, operation as the tracking unitcomprises operation as the tracking unit configured to associate aposition of the first target and a position of the second target witheach other and update the positions of the first target and the secondtarget.
 3. The simulator according to claim 2, wherein the processor isconfigured with the program to perform operations such that operation asthe tracking unit comprises operation as the tracking unit configured todetermine a size of the predetermined range from the surface of thesecond target in accordance with a height of the second target in thethree-dimensional virtual space.
 4. The simulator according to claim 1,wherein the processor is configured with the program to performoperations further comprising operation as an input unit configured toreceive a setting for a size of the predetermined range.
 5. Thesimulator according to claim 1, wherein in the three-dimensional virtualspace, an area indicating the predetermined range is expressed by usinga semitransparent object.
 6. The simulator according to claim 1, whereinthe processor is configured with the program to perform operations suchthat, in response to the processing device places the target within thepredetermined range from the transporting surface, operation as thetracking unit comprises operation as the tracking unit configured toupdate a position of the target in a height direction in thethree-dimensional virtual space to bring the target into contact withthe transporting surface.
 7. The simulator according to claim 2, whereinthe processor is configured with the program to perform operations suchthat, in response to the processing device placing the target within arange comprising the transporting surface and the second target,operation as the tracking unit comprises operation as the tracking unitconfigured to associate the target with one of the transporting surfaceand the second target based on predetermined priorities.
 8. Thesimulator according to claim 1, wherein the processor is configured withthe program to perform operations further comprising operation as ameasurement unit configured to perform image measurement of an inputimage comprising at least a part of the target as a subject of the inputimage associated with an area predefined on the transporting surface inthe three-dimensional virtual space, and the processor is configuredwith the program to perform operations such that, in response todetection of the target by the measurement unit, operation as thetracking unit comprises operation as the tracking unit configured todisplay the detected target in the three-dimensional virtual space.
 9. Asimulation method implemented by a computer for estimating a behavior ofa system comprising a carrier having a transporting surface forcontinuously transporting a target that is placed thereon and aprocessing device for picking up and placing the target onto thetransporting surface, the method comprising: virtually creating thesystem in a three-dimensional virtual space; updating positions of aplurality of targets placed on the transporting surface in thethree-dimensional virtual space based on a corresponding movement of thecarrier, and updating the position of the target picked up by theprocessing device in the three-dimensional virtual space in a mannerassociated with a behavior of the processing device; generating acontrol instruction for the behavior of the processing device based on aposition of each target of the plurality of targets in thethree-dimensional virtual space; and correcting the position of thetarget in a horizontal direction in the three-dimensional virtual spacein accordance with a transporting speed in the horizontal direction inthe three-dimensional virtual space at a timing in response to theprocessing device releasing the target, wherein updating the positionscomprises, in response to the processing device placing the targetwithin a predetermined range from the transporting surface, associatingthe target with the transporting surface and updating the position ofthe target.
 10. A non-transitory computer readable medium storing asimulation program for estimating a behavior of a system comprising acarrier having a transporting surface for continuously transporting atarget that is placed thereon and a processing device for picking up andplacing the target onto the transporting surface, the simulationprogram, when read and executed, causing a computer to implement:virtually creating the system in a three-dimensional virtual space;updating positions of a plurality of targets placed on the transportingsurface in the three-dimensional virtual space based on a correspondingmovement of the carrier, and updating the position of the target pickedup by the processing device in the three-dimensional virtual space in amanner associated with a behavior of the processing device; generating acontrol instruction for the behavior of the processing device based on aposition of each target of the plurality of targets in thethree-dimensional virtual space; and correcting the position of thetarget in a horizontal direction in the three-dimensional virtual spacein accordance with a transporting speed in the horizontal direction inthe three-dimensional virtual space at a timing in response to theprocessing device releasing the target, wherein updating the positionscomprises, in response to the processing device placing the targetwithin a predetermined range from the transporting surface, associatingthe target with the transporting surface and updating the position ofthe target.